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ABSTRACT 



A study on sampling errors of variance components was 
conducted within the framework of general izabi li ty theory by P. L. 
Smith (1978). The study used an intuitive approach for solving the 
problem of how to allocate the number of conditions to different 
facets in order to produce the most stable estimate of the universe 
score variance. Optimization techniques are proposed as a promising 
approach for solving allocation problems in generalizability studies. 
An important consequence of the conclusion that allocation problems 
in generalizability studies have much in common with allocation 
problems in decision studies would be a considerable reduction in the 
effort required to develop optimization procedures and computer 
programs. This examination of optimization techniques is the 
beginning of a procedure that enables the researcher to design a 
generalizability study in line with stated limits on sampling errors 
as well as resources. (SLD) 
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Abstract 

A major study on sampling errors of variance components was conducted within the 
framework of generalizability theory by Smith (1978). The study employed an intuitive 
approach for solving the problem of how to allocate the number of conditions to different 
facets in order to produce the most stable estimate of the universe score variance. In this 
study, optimization techniques are proposed as a promising approach for solving allocation 
problems in generalizability studies. 

Key words: generalizability theory, variance components, mathematical programming, 
sampling errors. 
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Introduction 

Generalizability theory (Cronbach, Gleser, Nanda, & Rajaratnam, 1972) makes a 
distinction between a generalizability study and a decision study. A generalizability study is 
conducted to obtain estimates of variance components associated with the universe of 
admissible observations. In a decision study these estimates are used to make decisions on 
the actual composition of a measurement instrument. Owing to limited resources, however, 
generalizability studies are often carried out with relatively small samples and relatively few 
conditions of each facet. Cronbach et al. (1972) have pointed out the danger of employing 
estimates of variance components and coefficients of generalizability studies based on small 
samples. The estimates of the variance components in these studies frequently entail sampling 
errors too large to be useful for a decision study. 

A major study on sampling errors of variance components was conducted within the 
framework of generalizability theory by Smith (1978). The study employed an intuitive 
approach for solving the problem of how to allocate the number of conditions to different 
facets in order to produce the most stable estimate of the universe score variance. In recent 
studies, the versatility of optimization techniques for solving allocation problems in decision 
studies was amply demonstrated. In this paper, optimization techniques are proposed as a 
promising approach for solving allocation problems in generalizability studies. 

Sampling errors of estimates of variance components are derived first. The 
minimization of sampling errors of variance components under constraints is discussed next. 
The balanced two-facet random-model crossed design will be used for illustration. 

Sampling Errors of Estimates of Variance Components 

In generalizability theory, the analysis of variance method (e.g., Searle, 1971, p. 384) 
is used for the estimation of variance components from balanced data. This method starts 
with calculating the mean squares of the analysis of variance, whereafter the expected values 
of the mean squares are derived under the random or mixed model. Table 1 contains the 
expected mean squares for the two-facet random-model crossed design. 
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TABLE 1 



Expected Mean Squares for p x i x 


j Random-Model Crossed Design 


ouurcc or variance 


Expected Mean Squares 


rersons [p) 


EMS p 


= o 2 pijie + n,a 2 pj + npW + n t n.a 2 p 


items 


EMS, 


= a liM + n P °l + "/pi + n p np] 


Raters (j) 


EMS. 


" °pW + + n i a lj + W; 


Persons x items (pi) 


EMS pi 


= 0 lij,e + "/pi 


Persons x raters (pj) 


EMS p . 


= °P«M + *A 


Items x raters (ij) 


EMS,. 


= a lij,e + n p4 


Residual (pij,e) 


EMS res 


2 

= °pij,e 



Equating the expected values to the calculated values leads to linear equations in the variance 
components, the solutions to which are the estimators of the variance components. The 
variance components for the two-facet random-model crossed design can be obtained with* the 
following equations: 



* 2 
°pij.e 


= MS piJ,e 






- 2 

°u 


= ( MS iJ ~ 


MS Pij,)ln p 




* 2 

°Pi 


= ( MS PJ - 


■ MS PiM)l»i 




* 2 

° P i 


= {MS pi - 


■ MS P<J,e)l»J 




* 2 

°j 


- ( MS J - 


MS,. - MS p . 


+ MS PiJ.e)t( n P n i) 


* 2 
°i 


= (MS, - 


MS,. - MS p , 


+ MS PiJ,)/M 


* 2 

°p 


- (MS p - 


MS pj ~ MS p , 





In the statistical literature (e.g., Searle, 1971, pp. 415-417), the theoretical sampling 
variance of estimates of variance components is expressed in terms of the expected mean 
squares. As was shown in Table 1, the expected mean squares are linear combinations of the 
population values of the variance components, and therefore the expected sampling variance 
of the variance components of the two-facet random-model crossed design can be expressed 
as: 
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The three factors affecting the magnitude of the sampling errors of the variance 
components can be inferred from equations (la-lg). 

The first factor concerns the relative magnitudes of the population values of variance 
components. While the population values of the variance components are not known, 
equations (la-lg) can be very useful for a researcher planning a generalizability study if prior 
information on the relative magnitude of the population values of variance components is 
available from comparable studies. In Table 2, for example, total variance is proportioned 
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in a way that resembles estimates commonly observed in applications of the two-;'acet design 
with respect to judgmental data such as observations or ratings (Smith, 1976, p. 90). The 
residual component is represented by a small (a 2 res = 20) and a large (a 2 res = 76) error 
component. 

TABLE 2 

Variance Components of a Two-Facet Random-Model Crossed Design 

Source of Variance Variance Components Proportion of total variance 
small error large error 



Persons (p) 


2 

°p 


« 20 


.09 


.07 


Items (i) 


2 
°« 


-.47 


.21 


.17 


Raters (/) 


2 
°j 


= 12 


.05 


.04 


Persons x items (pi) 


2 

°pi 


= 64 


.29 


.23 


Persons x raters (pj) 


2 

°PJ 


= 35 


.16 


.13 


Items x raters (if) 


2 
°V 


= 23 


.10 


.08 


Residual (pij>e) 


2 

°r« 


= 20,76 


.09 


.27 



The effect of the second factor, design configuration, on the magnitude of the sampling error 
of the universe score variance can be seen by comparing equation (la) to equation (2) for a 
design which has facet J nested within facet / : 



« 2 

var a p 



2 



2 N 



/ 2 



2 \ 



n h) J 



(2) 



The comparison of the two equations indicates that the sampling error of the estimate of the 
universe score variance can be decreased by employing a nested instead of a crossed design. 
The third factor concerns the number of measurement objects and the number of conditions 
of each facet. It is obvious that a researcher can avoid large sampling errors by employing 
large samples and large numbers of observations in a generalizability study. According to 
Smith (1978, p. 322), however, many studies use rather small sample sizes. In the designs 
examined in his study, the number of subjects (n p ) were represented at three magnitudes: low (n p 
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= 25), medium (n p = 50), and high (n p = 100). The number of conditions of each of the 
two facets (». and njj were represented at two, four, and eight conditions. 

Minimizing Sampling Errors of Estimates of Variance Components 

Three factors affecting the sampling errors of variance components were identified 
above. However, a researcher cannot influence the first factor, and the second factor implies 
changing the design of th; generalizability study. Therefore, given a certain design choice, 
a researcher can only manipulate the third factor. In the following, two approaches for 
controlling these sampling errors are presented. The results of the approach followed by 
Smith (1976, 1978) will be examined first. An approach for solving allocation problems 
based on mathematical programming is presented next. 



Intuitive Approach 

Under the constraints of a fixed number of n p subjects, and a fixed number of 
L = n^j observations, Smith (1978) sought the solution for the problem of how to allocate 
the observations in order to minimize the sampling variance of the universe score variance. 
Since, according to Smith (1978, p.328), simple solutions for this minimization problem 
could not be derived analytically, a more intuitive approach was chosen. From the 
examination of equation (la) he concluded that as the residual component increases, equal 
allocation will maximize the divisor of the residual component and the optimal solution tends 
toward n,. = n. . From a study by Woodward and Joe (1973) he borrowed the conclusion that 
as the residual component decreases, the allocation to n. and n, is detennined by the ratio 
of a pi and a pj . In order to evaluate these two conclusions, the minimization problem is 
stated in terms of mathematical programming as 



2 



minimize var o p objective-function (3) 

subject to L = tip. , and 

n p = constant. equality constraints (4) 

Objective-function (3) can be composed of four parts: 
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2 \ 
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(3c) 



/ 2 \2 

a.. 



(» r l)(«,-l) 



v^yy 



(3d) 



From equations (3a - 3d) it can be inferred that (3a), the 'persons' part, determines the value 
of the objective-function almost completely. With the appropriate variance components from 
Table 2 with a small error component and = 8 and n j = 4, the contribution of (3a) to the 
value of the objective-function is 97%. With a large error component, this contribution is 
96%. For both error components, the contribution of (3d), the 'residual' part, is negligible. 
The optimal continuous solutions for the variance components from Table 2 with L = 32 and a) es 
equal to 20 or 76, are computed as n t = 7 .4 and n y = 4.3. The same solutions are obtained 
wim °res e( l ua l t0 zero - With an improbable residual component equal to 200, however, the 
optimal continuous solutions are only slightly different, «. = 7.3 and n. = 4.4. Only an 
infinitely large residual component would affect the value of the objective-function and result 
in a solution tending toward n t = n y The same optimal integer solutions, n t = 8 and n } = 
4, are obtained with different residual components. The conclusion therefore is that the 
optimal number of conditions for the two facets does not depend on the magnitude of the 
residual component. To a lesser degree the same conclusion applies to parts (3b) and (3c), 
the 'interaction' components, of equation (la). 

Woodward and Joe (1973) showed that for the problem of maximizing the generalizability 
coefficient subject to the constraint that the total number of observations is fixed, the optimal 
allocation of conditions should be made directly proportional to o 2 pi /a 2 pj . In Sanders, 
Theunissen, and Baas (1991), their problem was stated in terms of mathematical 
programming as 
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2 2 2 

• • • °P< °PJ a res 

minimize — + -£ + objective-function (5) 

subject to L = «/i y . equality constraint (6) 



Because of equality constraint (6), the residual component does not play a role in detennining 
the optimal number of conditions. The optimal continuous solutions can be shown to be 



2\W 



"Pi 



J PJ 



, and n, 



2 

. °Pi) 



1/2 



Smith (1978, p. 329) fails to explain why the results obtained for the optimization problem 
defined by (5) and (6) also apply to a seemingly quite different optimization problem as 
defined by (3) and (4). The reason is that the dominant role of part (3a) in objective-function 
(3) makes the problem of minimizing objective-function (3) almost identical to the problem 
of minimizing objective-function (5). Note that by taking the square root of (3a) and leaving 
out the constant a pf equation (5) is obtained. The continuous solutions for the minimization 
problem defined by (3) and (4) will therefore hardly differ from those for the minimization 
problem defined by (5) and (6). The optimal continuous solutions for the optimization 
problem defined by (5) and (6) for the variance components from Table 2 are n, = 7.6 and 
tij = 4.2. 



Mathematical Programming Approach 

Mathematical programming can be used to solve the problem of controlling the 
sampling errors of variance components. The allocation problem is viewed as the 
minimization of sampling errors under constraints regarding observations and as the 
minimization of observations under constraints regarding sampling errors. 

From the foregoing discussion the conclusion can be drawn that for most 
generalizability studies the continuous solutions of optimization problems defined by (3) and 
(4) will not be much different from the continuous solutions of optimization problems defined 
by (5) and (6). A procedure for solving the optimization problem defined by (5) and (6) was 
presented in Sanders, Theunissen, and Baas (1991). This procedure, which also allows for 
less restrictive and more realistic constraints than (6), can be applied to the optimization 
problem under consideration. 
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Smith (1978, p. 327) only discusses the problem of minimizing the sampling error of 
the universe score variance. Mathematical programming, however, offers the possibility to 
solve optimization problems with multiple objective-functions. Williams (1985, pp. 24-26) 
describes three ways of dealing with multiple objective-functions. First, the optimization 
problem can be solved with each objective-function in turn. A satisfactory solution for the 
optimization problem can be selected from the solutions resulting from these different 
objective-functions. Second, objectives and constraints can be interchanged, that is, treating 
all but one objective-function as a constraint. Third, a linear combination of all the objective- 
functions and optimize the multiple objective-function can be taken. The optimization problem 

minimize w p var a] + w.var of + wjvar a) objective-function 
subject to L - nfij , and 



is a version of the optimization problem defined by (3) and (4) with (3) now being a linear 
combination of three ( bjective-functions. This solution to the optimization problem requires 
giving relative weights, w p , w., and w Jt to the three objective-functions. A less restrictive 
formulation of this multi-objective optimization problem is obtained by introducing an 
inequality constraint for the sample size, n p z 100. It can be shown that inequality 
constraints have many advantages over equality constraints. According to Williams (1985, 
p. 26), there is no one obvious way of dealing with multiple objectives. Some or all of the 
three methods should be used for certain optimization problems. 

The other approach for controlling sampling errors of variance components consists of 
minimizing the number of observations and/or the number of measurement objects under 
constraints of specific sampling errors. A general formulation of this optimization problem 
with one objective-function is 



n p = constant, 



equality constraints 



minimize n n n,n. 



objective-function 



subject to var a 2 p <. e 



threshold constraint 
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where e p stands for the value of the sampling error of the universe score variance considered 
to be acceptable. In order to gain an idea of the magnitudes of the sampling errors of 
variance components, Smith (1978, p. 335) suggested comparing the sampling errors obtained 
by equation (la) with those that would be expected for variance estimates obtained by 
drawing repeated samples from a single normal distribution. Assuming a fixed number of 
subjects, a restricted version of the foregoing optimization problem is 

minimize n.rij objective-function 
subject to var b 2 p z e p . • threshold constraint 



Assuming that the sampling error of the universe score variance is defined by equation (5), 
solutions for this optimization problem can be obtained by uie optimization procedure 
presented in Sanders, Theunissen, and Baas (1989). That procedure could also be used to 
obtain starting solutions for optimization problems with more complex objective-functions 
and/or constraints. 

An example of an optimization problem with multiple constraints is 



minimize n^ttj objective-function 

subject to var o 2 p < e p , and 
var o] <. e. , and 

- 2 

var Oj <; , threshold constraints 



where e p , e x , and vv y , stand for the values of three sampling errors of variance components. 
In general, the value for e p will be smaller than the value for e { or e. . Because the size of 
the sample is often the factor responsible for the major part of the costs of a generalizability 
study, the optimization problem could be formulated as 
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minimize np, } 



subject to n £ L 



objective-function 
threshold constraints 



var o p z e p , and 
var o] <> e n and 



var Oj £ e jy 

where L is the maximum sample size that the researcher can afford. 

Conclusions and Discussion 

In the study presented in this paper it was shown how our understanding of factors 
determining the sampling errors of variance components can be improved by viewing the 
problem of the optimal allocation of observations as an optimization problem. This approach 
leads to the important conclusion that allocation problems in generalizability studies have 
much in common with allocation problems in decision studies. An important practical 
consequence of this conclusion could be a considerable reduction of the effort required to 
develop optimization procedures and computer programs. 

Apart from the three factors discussed before, another factor affecting the magnitude 
of the sampling errors of variance components is the complexity of the design (Smith, 1981). 
The effect of this factor on the stability of the estimates can be shown by comparing equation 
(la) to the equation for a one-facet random-model crossed (p x i) design: 



var a_ = 

P (n-l) 



2 



2 \2 
*res 





( 2 ^ 


2 




1 








(n r l) 









From the comparison it is clear that the sampling error of the universe score variance 
component of the one-facet random-model will be smaller than the sampling error of the 
universe score variance of the two-facet random-model. However, there is an essential 
difference between the impact of this factor and the other three factors. Design complexity 
implies generalizations to different universes which means that the variance component 
estimates have different interpretations. 

The results of Smith's study (1978, p. 336) show that the sampling errors for the 
universe score variance are intolerably large for practical purposes unless n n t n. is 800 or 
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larger. Bell (1986) found the same results in a study on simultaneous confidence intervals of 
variance components. It should be clear to the researcher planning a generalizability study 
that the problem of sampling errors may make the results of the study unusable. Another 
problem is that of limited resources (time, money, etc.) which may render the study 
infeasible. What is needed therefore is a procedure which enables the researcher to design 
a generalizability study in line with stated limits on sampling errors as well as resources. The 
start of such a procedure was presented in this paper. 
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